On the Borel Inseparability of Game Tree Languages

1 Fa ulty of Mathemati s, Informati s, and Me hani s Warsaw University, Poland E-mail address: {shummel,henrykm,niwinski} mimuw.edu.pl Abstra t. The game tree languages an be viewed as an automata-theoreti ounterpart of parity games on graphs. They witness the stri tness of the index hierar hy of alternating tree automata, as well as the xed-point hierar hy over binary trees. We onsider a game tree language of the rst non-trivial level, where Eve an for e that 0 repeats from some moment on, and its dual, where Adam an for e that 1 repeats from some moment on. Both these sets (whi h amount to one up to an obvious renaming) are omplete in the lass of o-analyti sets. We show that they annot be separated by any Borel set, hen e a fortiori by any weakly de nable set of trees. This settles a ase left open by L.Santo anale and A.Arnold, who have thoroughly investigated the separation property within the μal ulus and the automata index hierar hies. They showed that separability fails in general for non-deterministi automata of type Σμn, starting from level n = 3, while our result settles the missing ase n = 2. Introdu tion In 1970 Rabin [15℄ proved the following property: If a set of in nite trees an be de ned both by an existential and by a universal senten e of monadi se ond order logi then it an also be de ned in a weaker logi , with quanti ation restri ted to nite sets. An automata-theoreti ounterpart of this fa t [15, 12℄ states that if a tree language, as well as its omplement, are both re ognizable by Bü hi automata ( alled spe ial in [15℄) then they are also re ognizable by weak alternating automata. Yet another formulation, in terms of the μal ulus [3℄, states that if a tree language is de nable both by a Πμ2 -term (i.e., with a pattern νμ) and a Σμ2 -term (μν), then it is also de nable by an alternation free term, i.e., one in Comp(Πμ1 ∪Σμ1 ). This last formulation gives rise to a question if the equation Πn ∩ Σ μ n = Comp(Π μ n−1 ∪ Σ μ n−1) holds on all levels of the xed-point hierar hy. Santo anale and Arnold showed [17℄, rather surprisingly, that it is not the ase for n ≥ 3. They exhibit a series of ambiguous properties, expressible by terms in Πn and in Σμn, but not in Comp(Πμn−1 ∪ Σμn−1). On positive side 1998 ACM Subje t Classi ation: F.1.1 Automata, F.4.1 Set theory, F.4.3 Classes de ned by grammars or automata.